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Abstract. We characterize the Sp2n-°rbits in the flag variety for SL^n with 
rationally smooth closure via a pattern avoidance criterion, also showing that 
the singular and rationally singular loci of such orbit closures coincide. 



1. Introduction 

Let G be a complex reductive group with Borel subgroup B and let K — G e 
be the fixed point subgroup of an involution of G. In this paper we continue the 
program begun in [M07 and continued in jMT08j , using pattern avoidance to char- 
acterize the A-orbits in G/B with rationally smooth closure (as in |LS90j ). Here 
we consider the case G — SL(2n, C), A — Sp(2n, C). We will adapt the techniques 
used in |E98j to study Schubert varieties for complex classical groups, focussing 
on the poset and graph structures of the set of orbits with closures contained in a 
given one. 

I would like to thank Axel Hultman for many very helpful email messages. 

2. Preliminaries 

Set G = SL(2n,C),K — Sp(2n,C). Let B be the subgroup of upper triangular 
matrices in G. The quotient G/B may be identified with the variety of complete 
flags Vq C V\ C ■ • • C Vm in C 2 " . The group A acts on this variety with finitely 
many orbits; these are parametrized by the set Ii n of involutions in the symmetric 
group Sa n without fixed points [MQ88, RS90 . In more detail, let (•, •) be the 
standard nondegenerate skew form on C 2n with isometry group A. Then a flag 
Vo C • • • C V~2n lies in the orbit corresponding to the involution 7r if and only if 
the rank of (•, •} on Vi x Vj equals the cardinality : 1 < k < i, 7r(fc) < j} for all 
1 < i,j < 2n. 

We will be using a modified version of the usual notion of pattern avoidance 
for permutations. We say that 7r = tti . . . 7T2„ (in one-line notation) includes the 
pattern /j = /ii . . . ^2m if there are indices i\ < i% < • • • < iim permuted by 7r 
such that TTi j > -Ki k if and only if [ij > /ifc (the usual definition would omit the 
condition that n permute the ij). We say that 7r avoids fi if it does not include 
it. For example, the involution 47513826 includes the pattern 351624: the indices 
1,2,4,6,7,8 are permuted by the involution and the first and third, fourth and 
sixth, and second and fifth of these are flipped. On the other hand, the involution 
78436512 does not include the pattern 3412, for although the indices 4, 6, 1, 2 occur 
in that order in the involution they are not permuted by it. 
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There are well-known poset- and graph-theoretic criteria for rational smoothness 
of complex Schubert varieties due to Carrell and Peterson C94j. These have been 
extended by Hultman to our setting (or more generally to -ftT-orbits in any flag 
variety G / B where the symmetric pair (G, K) corresponds to a real form Go of the 
reductive group G whose Cartan subgroups form a single conjugacy class [H09 ). 
To state them we first recall that the standard partial order on i^-orbits, by con- 
tainment of their closures, corresponds to the (restriction of the) reverse Bruhat 
order on l2 n |RS90j . The poset lm equipped with this order is then graded via the 
rank function 

r(rr) = n 2 - (ir(i) - i - #{fc 6 N : i < k < ir(k) < i}) 

Z<7r(z) 

where this quantity equals the difference in dimension between O n and O c , the 
unique closed orbit, corresponding to the involution wq = 2n . . . 1 RS90]. Let I n 
be the interval consisting of all tt' < tt in the reverse Bruhat order. Then Hultman 
has shown that is rationally smooth if and only if I n is rank-symmetric in the 
sense of having the same number of elements of rank i as of rank r — i for all i, 
where r is the rank of tt; equivalently, if and only if the rank generating function 
P n = J2^<n1 r ^' ) is palindromic [H09l 5.9]. If we make /„• into a graph BG n by 
decreeing that the vertices /i and v are adjacent if and only if v = tfit ^ jj, for some 
transposition t in Sam then O v is rationally smooth if and only BG^ is regular of 
degree r. If fx < n and we make the reverse Bruhat interval [/it, tt] into a graph 
BG/j.,ir by the same recipe, then 0„ is rationally smooth at if and only if the 
degree of \x in BG^^ is r{-n) — r(fi) H09J 4.5,5.8,6.7] (but in general BG Mi7r need 
not be regular or rank-symmetric in this situation) . In general the the degree of fx 
in BG^^ is always at least r := r(rr) — r(ji). We call /i an irregular vertex if it has 
larger degree than r. 

3. Main result 

We begin with a lemma about the inductive behavior of vertex degrees in Bruhat 
graphs. 

Lemma. Let (i, tt be two involutions in l2 n with fi < tt in reverse Bruhat order. 
Let t be a transposition of two indices flipped by both [i and it and set tt = jrt, fi = fit 
(so that 7r, fi are not in lm)- Let tt 1 , // be the unique involutions in Lin-i such that 
tt,(1 include the patterns 7r',/x', respectively, in the indices fixed by t. Assume that 
the vertex fjf is irregular in BG^i^i . Then /i is irregular in BG^ ^. 

Proof. Note first that the one-line notation of tt', for example, is obtained from that 
of tt by deleting the indices flipped by t and then replacing the ith smallest of the 
surviving indices by i. Thus if tt = 361542 and t flips 1 and 3, then tt' = 4321. We 
say that the transposition (a, d) flipping the indices a and d with a < d encapsulates 
the transposition (6, c) with b < c if a < b < c < d. Then the rank difference 
r(-7r) — r(/i) is given by r(Tr') — J"(//) + 2(n(p) — n(7r)), where n(p),n(Tr) are the 
numbers of transpositions in /i, tt, respectively, encapsulating t. Now every edge 
from // in BG^'^' corresponds to an edge from fi in BG^^ in an obvious way. For 
every transposition counted by n(/j,) but not n(rr) one easily locates two additional 
edges from /i in BG^.^, showing that /i is irregular whenever fi' is, as desired. □ 

Now we can characterize the i'C-orbits with rationally smooth closure. 
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Theorem 1. The orbit has rationally smooth closure if and only if it avoids the 
17 patterns 351624, 64827153, 57681324, 53281764, 43218765, 65872143, 21654387, 
21563487, 34127856, 43217856, 34128765, 36154287, 21754836, 63287154, 54821763, 
46513287,21768435. 

Proof. Note first that this list of bad patterns is stable under the automorphism of 
the Dynkin diagram: the first nine patterns are fixed by this automorphism while 
the next four pairs of patterns are interchanged. Suppose first that it coincides 
with one of the bad patterns. Then the bottom vertex in BG^ is irregular, as one 
sees from the following table. Here the rank of each bad pattern (regarded as an 
element of Iq or Is) is given in the middle column and the transpositions labelling 
the edges from the bottom vertex are given in the right column, abbreviating the 
flip of the ith and jth coordinates by ij . 



vertex 


rank 


edges from bottom vertex 


351624 


4 


12,13,14,23,24 


64827153 


5 


12,13,23,24,25,34,35 


57681324 


5 


12,13,14,23,24,34 


53281764 


7 


12,13,14,23,24,25,34,35 


43218765 


8 


12,13,14,15,23,24,25,26,34,35 


65872143 


4 


12,13,23,24,34 


21654387 


10 


12,13,14,15,16,17,23,24,25,26,34,35 


21563487 


11 


12,13,14,15,16,17,23,24,25,26,34,35 


34127856 


10 


12,13,14,15,16,23,24,25,26,34,35 


43217856 


9 


12,13,14,15,23,24,25,26,34,35 


34128765 


9 


12,13,14,15,16,23,24,25,26,34,35 


36154287 


9 


12,13,14,15,16,23,24,25,26,34,35 


21754836 


9 


12,13,14,15,16,23,24,25,26,34,35 


63287154 


6 


12,13,23,24,25,34,35 


54821763 


6 


12,13,23,24,25,34,35 


46513287 


8 


12,13,14,15,23,24,25,34,35 


21768435 


8 


12,13,14,15,23,24,25,34,35 



Now if 7r contains a bad pattern, then repeated use of Lemma 1 shows that 
BG^n is irregular at /i, where the one-line notation of /i is obtained from that 
of 7r by rewriting the indices in the bad pattern in decreasing order and leaving 
the other indices unchanged. Conversely, suppose that 7r avoids all patterns in the 
above list. We will show that the rank generating polynomial P n is palindromic, or 
more precisely it is the product of sums of the form 1 + q + ■ ■ ■ + q l for various t . Let 
7r = 7Ti . . . 7r2 n and assume first that In — w± < ~Ki n — 1 (i.e., 1 is closer to the end of 
tti . . . 7T2n than 2n is to the beginning). Set tt 1 -- 1 ^ — tirt, where t is the transposition 
interchanging tti and 7Ti + 1, so that 1 appears one place further to the right in 
tt^ than in 7r. Define n^ 2 \ . . . ,7r( 2n_ ' Tl ) similarly, so that 1 appears at the end of 
^(271-tti) If [/, = . . . [i 2n < 7r then Proctor's criterion for the Bruhat order |P82j 
shows that [i\ > tti. If /ii = iti, then one checks that // < n' , where fJ,',Tv' are 
obtained from /z, tt by omitting the indices 1 and /xi, replacing all indices i between 
1 and /zi by i — 1, and replacing all indices j > \i\ by j — 2; moreover, tt' continues 
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to avoid all bad patterns. If instead fii > 7ri, then we claim that /i < tt^ and that 
tt^ continues to avoid all bad patterns. If this holds, then induction shows that 
/j, < ■k^ 1 —* 1 ') whence we may as above eliminate the indices 1 and fi\ from \x and 
^(^i-ti) anc j re p ea t the above procedure. Using the formula for the rank function 
in I 2n , we deduce that P^ factors in the way claimed above, where the first factor 
is 1 + gH ^q 2n - ni . 

To prove the claim, set tt\ — k, Tik+i = i and suppose that there is [i with 
yu < 7r,^ ^ 7rW, and yn > tt\. Set j :— [i],. There are two cases. If i < k, then 
the conditions [i < tt,^l ^ tt^ force ttj > k + 1, whence tt contains the pattern 
p := 465132 (since it avoids the pattern 351624). The assumption 2n — tt\ < 7r 2 „ — 1 
implies that tt ^ p, so that tt is the product of three transpositions forming the 
pattern p and at least one more transposition. Now one checks that no matter how 
one chooses this transposition to force 2n — tt\ < 7r 2n — 1 we get a bad pattern 
in tt, a contradiction; more precisely, one of the five patterns 46513287, 63287154, 
65872143, 64827153, or 57681324 must occur in tt. Similarly, if instead i > k + 1, 
then one must again have irj > k + 1. In this case tt contains the pattern 361542, 
and once again the assumption 2n — tt\ < TX2n ~ 1 forces it to contain a bad pattern. 
Here the bad patterns that arise are 36154287 and 53281764. 

If instead 7r 2n — 1 < 2n — tti, then one repeats the above argument, replacing 1 
by 2n and moving 2n to the left instead of 1 to the right. Thus we define tt^\ tt^ 2 \ 
and so on, so that 2n appears one place to the left in tt^ than it does in tt; if 
a < tt then we must have u 2n < 7r 2 „, and if u 2n < T2n, then we must have /x < 7rW, 
lest 7r contain a bad pattern. Here the two "bad seeds" that must be ruled out 
are 546213 and 532614; these give rise to the bad patterns 21768435, 54821763, 
65872143, 64827153, 57681324, 21754836, and 53281764. 

Finally, we must ensure in both cases that tt^ avoids all bad patterns whenever tt 
does. This requires that we rule out four more "bad seeds" , namely 216543, 432165, 
215634, and 341265; we achieve this by ruling out the bad patterns 21654387, 
43218765, 34127856, 43217856, 21563487, and 34128765. Excluding also the bad 
pattern 351624 of length 6, we see that if tt avoids all bad patterns then factors 
in the desired way and O n has rationally smooth closure, as required. □ 

4. Smoothness and the bottom vertex 

We now consider reverse Bruhat intervals [n, tt] and their graphs 5G MjT . We will 
find (as for Schubert varieties in type A) that it is only necessary to test one vertex 
in this graph to determine whether or not O n is (rationally) smooth at O^. 

Theorem 2. If u < tt and the degree of u in BG^^ equals r{ir) — r(/z), then 
is smooth along O. In particular, the singular and rationally singular loci of O 
coincide. 

Proof. Assume first that C M = Cl the closed orbit. Fix a basis (e^) of C 2 " such 
that 

{1 if i < j, i + j = 2n + 1 
-1 if i > j, i +j = 2n+ 1 
otherwise 

where (•, •) is the skew form. Let (ay) be a family of complex parameters indexed 
by ordered pairs (i, j) satisfying either i < n < j or n < i < j. We assume that 
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i_i and di^n+i-i = if i < n but otherwise put no restrictions 



a-ij — —d2n+l-j,2n+ 

on the aij. Define a basis (bi) of C 2 ™ via 



2ra 

£ 

2n 



dij-ej if i < n 



*tj*ej otherwise 



j=»+i 

Then the Gram matrix G := (gij = ((6,, &;))) of the bi relative to the form satisfies 

2aj,2n+i-j if i < j < n 

~9ji if 3 < i < n 

if i < n < j < In + 1 — i 
1 if i < n < j = 2n + 1 — i 

~9ji if j < n < i 

otherwise 

Thus the matrix G is skew-symmetric and has zeroes below the antidiagonal from 
lower left to upper right. The antidiagonal entries are all ±1. Now one checks that 
the set T of all flags Vo G ... C V% n where (pi) runs through all bases obtained as 
above from the and Vi is the span of &i, . . . bi is a slice in the sense of Brion to 
O c at the flag f c corresponding to the basis (e^) [Br991 2.1]. Intersecting T with 
O-K we get another slice to O c at f c . 



By hypothesis there are n 



r(n) distinct conjugates c = twot of wo by a 



transposition t such that c ^ n. One computes that d :— two also satisfies d j£ it. 
Writing d as di . . . c?2n , let i be the smallest index such that iri . . . %i d\ . . . di 
in the standard partial order on sequences used to characterize the Bruhat order 
[P82j . Thus if 7Ti . . . 71"$ is rearranged in increasing order as 71^ . . . tt[ and similarly 
d\ . . . di is rearranged as d[ . . .d' i: then tt'j > d'- for some j. Then for some k there 
are more indices I < i with dg < k than indices m < i with 7r m < fc. Equating all 
minors of the appropriate size lying in the first i rows and columns d\ , . . . , di of the 



Gram matrix to 0, we arrive at n 



r(jt) polynomials vanishing on n 



each involving a distinct variable raised to the first power with coefficient ±1. Then 



the Jacobian matrix of these polynomials has rank n 



r(n), whence by the 



Jacobian criterion both and are smooth at O c , as desired [Br99, 2.1] 

If O c is replaced by any orbit C M C O n , then let G^ be the matrix whose ij- 
entry is 1 if j = fj,j > i, —1 if j = //j < i, and otherwise. This is the Gram 
matrix of a basis (bi) obtained by suitably rearranging the basis (e^); let / M be the 
corresponding flag. Now consider the set of all Gram matrices G whose ij-entries 
agree with those of G M if j > fii and whose other possibly nonzero entries are 
determined as follows. There are n 2 — n — r(/x) conjugates c of \i by transpositions 
with c > fx. Write each c as c\ . . . ci n and let i be the smallest index with Ci < /ij. 
Then the other possibly nonzero entries of G appear in the positions (i, Cj) together 
with their transposes (ci,i). Entries of G not in one of the positions specified above 
are 0. There are no further restrictions on these entries apart of course from being 
skew-symmetric. This set of Gram matrices stands in bijection to a set J 7 ^ of flags 
which is a slice to at / M . Then one shows as above that if the degree of [i in 



BG^^ equals the difference r(n) — r(n), then there are n 



r(ir) polynomials 
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vanishing on f) J-^ whose Jacobian matrix has rank n 2 — n — r(ir), whence again 
Ott is smooth along M , as desired. □ 

There are two other symmetric pairs (G, K) of complex reductive groups satis- 
fying the hypothesis of H09J (that all Cartan subgroups in the corresponding real 
form Go of G are conjugate), namely (Spin(2n, C), Spin(2n — 1,C)) and {Eq,F^). 
In the first case all i^-orbits in G/B have smooth closure. In the second case, 
eleven out of the forty-five -ftf-orbits have rationally singular closure. Hultman has 
checked in each case that the bottom vertex of the Bruhat graph detects the ratio- 
nal singularity. It is not known whether smoothness and rational smoothness are 
equivalent for these orbit closures. 
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